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ABSTRACT 

Objective: To characterise clinical questions raised by 
providers in the care of complex older adults in order 
to guide the design of interventions that can help 
providers answer these questions. 
Materials and methods: To elicit clinical questions, 
we observed and audio recorded outpatient visits at 
three healthcare organisations. At the end of each 
appointment, providers were asked to identify clinical 
questions raised in the visit. Providers rated their 
questions based on their urgency, importance to the 
patient's care and difficulty in finding a useful answer 
to. Transcripts of the audio recordings were analysed 
to identify ageing-specific factors that may have 
contributed to the nature of the questions. 
Results: We observed 36 patient visits with 10 
providers at the three study sites. Providers raised 70 
clinical questions (median of 2 clinical questions per 
patient seen; range 0-12), pursued 50 (71%) and 
successfully answered 34 (68%) of the questions they 
pursued. Overall, 36 (51%) of providers' questions 
were not answered. Over one-third of the questions 
were about treatment alternatives and adverse effects. 
All but two clinical questions were motivated either 
directly or indirectly by issues related to ageing, such 
as the normal physiological changes of ageing and 
diseases with higher prevalence in the elderly. 
Conclusions: The frequency of clinical questions was 
higher than in previous studies conducted in general 
primary care patient populations. Clinical questions 
were predominantly influenced by ageing-related 
issues. We propose a series of recommendations that 
may be used to guide the design of solutions to help 
providers answer their clinical questions in the care of 
older adults. 



INTRODUCTION 

In a seminal study, Covell et al 1 observed that 
physicians raised two questions for every 
three patients seen in an outpatient setting. 
In 70% of the cases, these questions were not 
answered. Recent research has produced 
similar results, with little improvement in the 
three decades since Covell's study was pub- 
lished. According to a systematic review, 



Strengths and limitations of this study 



This is the first study to observe clinical ques- 
tions in the care of complex older adults. 
Our method included direct audio recorded 
observations of providers in multiple phases of 
outpatient care. This method allows more 
detailed and accurate data collection, since it 
relies on direct observations of care as opposed 
to provider's recall. 

The study findings raise important implications 
to improve the design of online health knowledge 
resources and electronic health record systems. 
Direct comparisons of question frequencies were 
not possible because we did not observe clinical 
questions in non-ageing and non-complex 
patients. 

The small number of sites and providers in each 
subgroup precluded a comparison of questions 
between different setting types and provider 
types. 



estimates ranged from 0.2 to 1.9 clinical 
questions per patient seen, with less than 
half of these being pursued, and over 60% of 
questions not being answered. 2 Unanswered 
clinical questions may represent knowledge 
gaps that have been associated with errors 
and reduced quality of care. 3 This problem 
may be aggravated by the increasing volume 
of medical knowledge and patient complex- 
ity, especially associated with the ageing 
population. 4-6 

The number of older adults in our society 
is increasing dramatically as the 'Baby 
Boomers' start to age. In addition, the 
number of geriatricians available to care for 
them is not keeping pace with the increase. 
In fact, family physicians provide the major- 
ity of care for older adults, 7 making educa- 
tion of these providers an important 
component of any programme to improve 
the quality of care. Caring for older adults is 
complex. Recent reviews assessing the 
quality of care provided for older adults 
have found significant deficits. For example, 
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researchers found that only half of the vulnerable 
elderly living in the community received care that met 
quality indicators and only a third received care for 
those conditions that primarily impact the elderly 8 In 
another recent review, Askari et af found rates of 
appropriate care to be variable across studies and very 
low for many geriatric-related conditions, including 
dementia (11-35%), depression (27-41%) and osteo- 
porosis (34-43%). 

Despite substantial previous research on providers' 
clinical questions, little is known about the specific 
characteristics of questions that arise in the care of 
ageing and complex patients. Knowledge of clinical 
questions in this patient population may be used to 
guide the design of interventions that help providers 
answer their questions and improve the care of older 
patients. The overall aim of this study was to address this 
gap. Specifically, we aimed at answering the following 
study questions: (1) How frequently do providers raise, 
pursue and answer their clinical questions? (2) How 
urgent, important to the patient's care and difficult to 
find an answer to are these clinical questions? (3) What 
types of questions are most commonly raised? (4) How 
often are these questions specific to geriatrics? (5) How 
do issues related to ageing affect these questions? 

METHODS 

Study participants and sites 

All study participants reviewed and signed an informed 
consent to participate in the study. We recruited 10 experi- 
enced geriatricians, family physicians and nurse practi- 
tioners from outpatient settings at three healthcare 
organisations located in Utah: a geriatric clinic at the 
University of Utah, a geriatric clinic at the Salt Lake City 
Veterans Administration Medical Center (\AMC) and a 
community clinic at Intermountain Healthcare 
(Intermountain) . We asked providers to identify complex 
patients who were scheduled for a visit during a typical 
clinic day. Complex patients were defined according to the 
Agency for Health Research and Quality (AHRQ) defin- 
ition as those with "two or more chronic conditions where 
each condition may influence the care of the other condi- 
tion (s) through limitations of life expectancy, interactions 
between drug therapies, and/or direct contraindications 
to therapy for one condition by other conditions 
themselves." 10 

Observations 

We focused on clinical questions as defined by Ely 
et at x \ "questions about medical knowledge that could 
potentially be answered by general sources such as text- 
books and journals, not questions about patient data 
that would be answered by the medical record." To elicit 
clinical questions, we conducted patient care observa- 
tions following the cognitive work analysis method, 
which is a group of techniques that integrate observation 
and interview for the purposes of understanding the 



constraints, resources, behaviour and cognitive goals of a 
work situation. 12 A researcher (AIW) observed and 
audio recorded providers in all activities related to a 
patient visit, including preparing for the visits (eg, 
reviewing the patient's chart), interacting with the 
patient and concluding the visit (eg, documentation, 
medication prescription) . Providers were asked to briefly 
summarise the case, listing the patient's problems, medi- 
cations and visit goals. At the end of each appointment, 
providers were interviewed regarding the clinical ques- 
tions that were raised in the visit. For each question iden- 
tified, we asked the provider to rate its importance and 
urgency; level of confidence in the clinical domain of 
the question (eg, treatment of depression) and the level 
of difficulty of finding an answer to. These measures 
were obtained using a Likert scale format for the ques- 
tionnaire. We also observed whether the question was 
pursued, asked providers whether a satisfactory answer 
was found and observed which information resources 
were used to answer it. The researcher contacted provi- 
ders for a follow-up interview about questions that were 
not answered in the visit within 4 weeks following the 
observation session. 



Data analysis 

Audio recordings were transcribed and de-identified for 
analysis. Two investigators (GDF, CRW) independently 
reviewed the transcripts to identify clinical questions. We 
identified questions that were both explicitly stated by 
providers in the postvisit interview and inferred from 
providers' verbalisations and observed information- 
seeking behaviour. Next, annotations were compared 
assisted by the researcher who conducted the observa- 
tions, and discrepancies were resolved by discussion 
until the investigators reached consensus. The final set 
of questions was coded independently by two investiga- 
tors (GDF, AIW) according to Ely's taxonomy of clinical 
questions. 11 In this phase, disagreements were also 
resolved by consensus. 

Clinical questions were also coded in terms of the 
degree to which ageing-related factors contributed to a 
question. An ageing factor was defined as a patient char- 
acteristic that is exclusive to, or more common in, 
ageing patients and that motivates or modifies the 
nature of a clinical question. Factors were identified, 
and questions were coded using the constant compari- 
son method. 13 In the first round, the four study authors 
independently proposed candidate factors for a subset 
of 20 questions. Next, the factors proposed by each 
investigator were reconciled through group consensus 
(one of the authors is an experienced geriatrician). In 
the second round, investigators used the set of recon- 
ciled factors to code another set of 35 questions. In this 
round, new factors were proposed and the definition of 
previous factors was refined through group consensus. 
In the third and final round, investigators coded the 
remaining questions resolving disagreements by 
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consensus. No changes to the factors were necessary in 
this final round. 

Finally, we conducted univariate analyses to test the 
association between urgency, importance, provider confi- 
dence and time pressure as predictors for the decision 
to pursue. Statistical significance was tested with the 
Fisher's exact test. We also assessed the association 
between number of questions per patient and number 
of questions pursued. Statistical significance was tested 
with analysis of variance, with the binary decision to 
pursue as the grouping variable. 



RESULTS 

Frequency of clinical questions raised, pursued 
and answered 

Providers raised 70 clinical questions in 36 patient visits 
(average of 1.9 questions per patient seen; median of 2 
questions per patient seen; range 0-12 questions), 
pursued 50 (71%) and successfully answered 34 (68%) 
of the questions they pursued. Most questions were 
pursued during the visit versus the follow-up period (48 
vs 2 out of 50 questions pursued). Overall, 36 (51%) of 
providers' clinical questions were not answered. 

Importance, urgency, confidence and difficulty 

Providers considered 42% (mean rating=3.0; l=not 
urgent; 5=very urgent) of their questions to be urgent or 
very urgent, and 81% (mean rating=4.1; l=not import- 
ant; 5=very important) to be important or very import- 
ant for the patient's care. Of the questions that were left 
unanswered, 45% were considered to be important or 
very important and 8% were considered to be urgent or 
very urgent. In 61% of the questions (mean rating=3.8; 
l=not confident; 5=very confident), providers felt that 
they were confident or very confident regarding their 
overall knowledge in the domain of the question. 
Providers perceived that only 14% (mean rating=2.2; 
l=not difficult; 5=very difficult) of the questions they 
pursued were difficult or very difficult to find an answer 
to. None of the associations between the independent 
variables (urgency, importance, provider confidence and 
time pressure) and a question being pursued were sig- 
nificant (table 1). Physicians were more likely to pursue 



Table 1 Association between urgency, importance, 
provider confidence and time pressure as predictors for the 



decision to pursue a clinical question 




Fisher's 


Degrees 






exact 


of 




Predictor 


test 


freedom 


p Value 


Urgency 


0.54 




0.64 


Importance 


0.37 




0.65 


Provider confidence 


0.99 




0.36 


Time 


2.2 




0.34 



questions for patients whose care generated a larger 
number of questions (F(l,68)=4.076; p=0.047). 

Types of clinical questions and ageing factors 

Table 2 shows the frequency of clinical questions accord- 
ing to Ely's taxonomy comparing to five previous studies 
that used the same taxonomy. Over one-third of the 
questions were about treatment alternatives and adverse 
effects. Most questions (68 out of 70; 97%) were directly 
or indirectly related to 1 of 10 ageing-specific factors 
(table 3). Over half (40; 57%) of the clinical questions 
were related to treatment factors, specifically treatment 
choice (18; 26%), prescribing considerations (13; 19%) and 
managing side effects (9; 13%). Table 3 proposes a set of 
recommendations to guide the design of online knowl- 
edge resources and electronic health record (EHR) 
systems in light of the ageing factors listed in table 3. 

DISCUSSION 

We characterised the clinical questions raised by provi- 
ders in the care of complex older adults. We found that 
providers raised three times more questions (1.9 vs 0.6 



Table 2 Clinical questions classified according to Ely's 
taxonomy and compared with pooled data from five 
previous studies 



Question type 



Previous 
studies (%) 



Current 
study (%) 



What is the drug of choice 
for condition x? 
What is the cause of 
symptom x? 

How should I treat condition 

x (not limited to drug 

treatment)? 

What is the cause of 

physical finding x? 

What test is indicated in 

situation x? 

What is the dose of drug x? 
Can drug x cause (adverse) 
finding y? 

What is the cause of test 
finding x? 

Could this patient have 
condition x? 
How should I manage 
condition x (not specifying 
diagnostic or therapeutic)? 
What is the prognosis of 
condition x? 

What are the manifestations 
of condition x? 
What conditions or risk 
factors are associated with 
condition y? 



10 



10 



16 



4 
13 



The data include the 13 most frequent question types that 
accounted for 80% of the questions asked across studies. 
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Table 3 Frequency of clinical questions per ageing factor 



Aging factor 



Frequency Definition 



Examples 



Special considerations 18 (26%) 

when choosing optimal 

treatment 



Special prescribing 13 (19%) 

considerations 



Complex management of 9 (13%) 
side effects 



Condition prevalence 



Understanding other 
provider's rationale 



Difficult diagnosis 



No ageing factor 



8 (11%) 



6 (9%) 



Dx testing considerations 4 (6%) 



Access to health services 4 (6%) 



4 (6%) 



Gender considerations 1 (1%) 



Need for geriatric tool 1 (1%) 



2 (3%) 



Selection of an optimal individualised 
treatment considering ageing factors such as 
risk/benefit and comorbidities. Successful 
outcome is more difficult because of 
underlying ageing issues 
Medication prescription needs to be adjusted 
to maximise compliance, and minimise side 
effects/organ damage (eg, by adjusting 
medication dose) 

Consideration of side effects. Issues such as 
polypharmacy and lower medication tolerance 
contribute to a higher incidence of and more 
complexity in managing side effects 

Condition related to the questions is much 
more prevalent in the elderly. Questions 
related to these conditions would be less 
common in non-ageing patients 
Unable to interpret rationale of other providers 
due to lack of enough information (eg, 
prescription without reason, diagnosis without 
explanation). Complex ageing patients are 
often cared for by multiple providers 
Ageing risk factors need to be considered in 
the choice of diagnostic intervention 

Health services that are more commonly 
needed or that have special requirements in 
elderly patients 

Difficult diagnosis due to underlying ageing 
factors (eg, multiple comorbidities, different 
presentation). Difficult to interpret new set of 
symptoms/signs/findings in light of the overall 
patient's picture 

Decisions in the elderly that are affected by 
gender (eg, different statin dose, different 
osteoporosis treatment) 
Need for tools (eg, assessment tools) that are 
specific for geriatrics 

Question not motivated or mediated by ageing 
and answer is not ageing-specific 



What is the preferred A1c goal in 
the ageing population? 
What is the best treatment choice 
for diabetes when the patient also 
has heart failure? 
What is the geriatric dose of 
buspar for depression? 
What is the CrCI cut-off for 
alendronate? 

Is hallucination a side effect of 
rivastigmine? 

Is there adjunct treatment of 
depression that does not cause 
drowsiness? 

What is the best treatment choice 
for cognitive dysfunction? 



What are these eye drops used 
for? 

What are the indications of 
concomitant use of aspirin and 
warfarin? 

Is contrast indicated for chest 
X-ray to assess aspiration in a 
patient with GERD? 
Where should I refer this patient 
for mental health? 

Why is this patient osteopaenic? 
What is the cause of this patient's 
weight loss? 

How do I manage cardiovascular 
risk in elderly women? 

Where can I find a template for 

haematology-oncology 

assessment 

Where can I find patient education 
information on cholesterol diet? 



GERD, gastroesophageal reflux disease. 



questions per patient seen) than in previous studies not 
focused on complex ageing patients. This higher rate of 
questions may be attributed to the complexity of patients 
seen and to ageing factors. We also identified a set of 
ageing-specific factors that motivated or affected most of 
the questions. These factors can be used to guide the 
design of solutions that can answer these questions more 
directly. 

Our study has a few important strengths. This is the first 
study to observe clinical questions in the care of complex 
older adults. Investigating these questions is important 
because the ageing population is rapidly increasing 5 and 
elderly patients with multiple comorbidities are more 



difficult to manage with available clinical practice guide- 
lines, 4 which lead to significant deficits in the quality of 
care. 8 9 14 As a second strength, our method included direct 
audio recorded observations of providers in multiple phases 
of outpatient care. Most previous studies elicited clinical 
questions in after-visit interviews or relied on providers to 
keep their own record of their questions. 2 Our method 
allows more detailed and accurate data collection, since it 
relies on direct observations of care as opposed to provi- 
der's recall, which could involve a possible bias. 

Over half of the questions raised in our study were left 
unanswered and providers rated close to half of these 
questions as important or very important for the 
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Table 4 Ageing factors and implications for the design of online knowledge resources and electronic health record (EHR) 
systems 



Ageing factor 



Implications for design 



Examples 



Special considerations when 
choosing optimal treatment 
and diagnostic testing 



Special prescribing 
considerations 



Complex management of side 
effects 



Understanding other 
provider's rationale 



Access to health services 



Online knowledge resources could provide 
specific recommendations to help providers 
tailor treatment and choose diagnostic tests 
considering ageing issues such as risk/ 
benefit, comorbidities, functional status and 
social support. These recommendations 
should be easily accessible/filtered by the 
resource's search engine based on the 
patient's age 

EHR systems should capture patient's life 
goals and integrate them into the patient's 
treatment plan 

Online knowledge resources could provide 
seamless access to age-specific guidance 
on dose adjustment, adherence issues in 
older adults and ageing-specific 
contraindications 
EHR systems could propose and 
automatically calculate adjusted medication 
dosing when indicated due to ageing factors 



Based on a patient's side effects and current 
medications, online resources could provide 
likely side effects for combinations of 
medications often seen in older patients. 
Online resources could automatically 
construct a side effect profile based on the 
medications documented on the patient's 
EHR. In addition, online resources could 
enable providers to simulate alternate 
medication scenarios and compare side 
effect profiles of alternate scenarios 
Providers should be able to document the 
rationale for their decisions (eg, prescribing a 
medication, discontinuing a medication, 
ordering a diagnostic test) in the patient's 
EHR and link the rationale to the decision. 
This documentation should support 
identification of how the provider addressed 
patient preferences, social support and 
functional status 

Based on a location of interest and the 
patient's age, the EHR could automatically 
link to information on health services 
available in the area 



"What is the preferred A1c goal in the ageing 
population?" 

Provide recommendations on how to adjust 
the A1c goal given factors such as the 
patient's age, preferences and life 
expectancy 

"What is the best treatment choice for 
diabetes when the patient also has heart 
failure?" 

Provide treatment recommendations in the 
presence of most common comorbidities 

"What is the geriatric dose of buspar for 
depression?" 

Allow the user to provide the patient's age in 
the search process and highlight the geriatric 
dose in the user interface. When prescribing 
a medication or reviewing a patient's 
medications list, display an icon next to a 
medication that is potentially inappropriate for 
ageing patients. Hovering the mouse over 
this icon provides an explanation and a 
suggested alternative 
"Which of the patient's medication may be 
causing hallucination?" 
Rather than scanning the list of side effects 
for each of the patient's current medications, 
EHRs could automatically send the side 
effect and the patient's medications list to 
online knowledge resources, which would 
return a table with the medications and their 
likelihood of causing the side effect of 
interest 

"What are the indications of concomitant use 

of aspirin and warfarin?" 

When hovering over a medication in the 

patient's medication list, the EHR shows the 

rationale of the prescriber for prescribing the 

medication 



"Where should I refer this patient for mental 
health?" 

A link from the EHR could automatically 
retrieve mental health facilities within the 
patient's location 



patient's care. These unanswered questions may contrib- 
ute to issues that disproportionally affect the elderly 
population, such as increased adverse events, 6 15-20 
inappropriate medication prescription, treatment failure 
and adverse drug withdrawal events. 14 

Consistent with previous studies, providers did not 
pursue over half of their questions. When providers 
pursued a question, they were successful most of the 



time. This might be an indication that providers self- 
select questions that can be answered with little effort. 
In our study, providers perceived that only 14% of the 
questions pursued were difficult to answer. Providers 
were more likely to pursue questions for patients whose 
care generated a larger number of questions. It is pos- 
sible that these patients were more complex and there- 
fore required more careful deliberation. 
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Compared with previous studies, we found a higher 
frequency of questions related to treatment alternatives 
and adverse effects. This finding could be explained by 
the presence of ageing-specific factors that motivated or 
affected nearly all questions observed in our study 
These factors commonly constrain or alter treatment 
choices, making treatment decisions more complex and 
often not amenable to available evidence-based guide- 
lines. 4 This is consistent with a study by Merten et al, 18 
which found the inability to apply existing knowledge to 
a new and complex situation to be an important con- 
tributor to adverse events in older patients. Providers in 
our study were often faced with the need to personalise 
treatment goals according to individual factors, such as 
undesired effects of treatment, comorbidities, patients' 
priorities and life expectancy. As healthcare delivery 
systems strive to provide patient-centred care, the need 
to personalise and integrate a patient's specific context 
will become increasingly important. 

Potential solutions 

As suggested in table 4, ageing-specific factors should be 
considered in the design of online knowledge resources 
and EHR systems. The design considerations provided in 
table 4 are technically feasible and international stan- 
dards are available to enable automated links between 
EHR systems and online knowledge resources. 21 These 
standards are being widely adopted in the USA as a 
requirement for EHR certification. 22 

Since providers rarely pursue questions after a 
patient's visit, solutions need to provide answers to provi- 
ders' questions rapidly, ideally in less than a minute. Yet, 
in a healthcare environment where providers spend on 
average 15 min/patient visit, 23 24 constraining informa- 
tion seeking to the time frame of a patient encounter 
may limit providers to pursuing easier questions. One 
alternative is to design interventions that help providers 
record their questions and pursue them at their conveni- 
ence. Answers to these questions could be automatically 
stored in the patient's EHR and shared with other provi- 
ders who manage similar patients through technologies 
like social media and recommender systems. In addition, 
automated analysis of recorded questions could be used 
to help providers define their lifelong learning goals as 
a component of Maintenance of Certification. 25 26 This 
form of self-directed learning could be more effective 
and compatible with the adult learning style than trad- 
itional forms of continuing medical education. 26 27 

Online knowledge resources could be designed to go 
beyond reporting of individual studies, and to supporting 
simulations of combinations of complex variables. A high 
level of integration is required in order to individualise or 
for tailor treatment, but few single studies address any spe- 
cific combination of risk, patient preferences, expected 
life expectancy and comorbidities. This requirement is not 
only needed in the older population, but also in other 
areas, such as children with special needs, immigrant 
populations and other unique populations. 



Limitations 

We did not observe clinical questions in non-ageing and 
non-complex patients. Therefore, direct comparisons of 
question frequencies were not possible. The small 
number of sites and providers in each subgroup, along 
with the presence of several potential confounders, pre- 
cluded a comparison of questions between different 
setting types (eg, academic vs community clinic) and pro- 
vider types (eg, family physicians vs geriatricians, nurse 
practitioners vs physicians). As in previous similar studies, 
the presence of an observer may have stimulated ques- 
tions and information-seeking behaviour. To minimise 
this risk, we observed providers in their typical busy 
routine as unobtrusively as possible, and asked them to 
carry out their work as they would normally do. In add- 
ition, observation studies have provided more reliable 
results than other methods, such as self-report and 
surveys, which are prone to recall bias. 2 Finally, the 
4-week time frame for follow-up may have introduced 
recall bias, as in previous studies most providers pursued 
their clinical questions within 24 h of a patient 
encounter. 2 

Future studies 

Studies are needed to design and assess interventions 
that help providers' decision-making in ageing and 
complex patients. As suggested in the previous sections, 
our findings provide important insights for intervention 
design. Moreover, larger studies are needed to enable 
subgroup comparisons such as the ones described 
above. 

CONCLUSION 

We found that providers raised a large number of clin- 
ical questions in the care of complex older adults and 
half of these questions were not answered. Compared 
with previous studies in younger adults, clinical ques- 
tions in the care of the older population were raised 
three times more often. We also found a relatively 
higher rate of questions related to treatment alternatives 
and adverse effects. Most of the questions were moti- 
vated or mediated by factors specific to ageing. When 
unanswered, these questions may contribute to issues 
that are more prevalent in the elderly, such as an 
increased rate of adverse drug events. Our findings may 
be used to help guide the design of information delivery 
interventions that help providers answer their clinical 
questions in the care of older adults. 
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